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Abstract. The transfer operator for Tq(N) and trivial character 
Xo possesses a finite group of symmetries generated by permuta- 
tion matrices P with P 2 = id. Every such symmetry leads to a 
factorization of the Selberg zeta function in terms of Frcdholm de- 
terminants of a reduced transfer operator. These symmetries are 
related to the group of automorphisms in GL(2,Z) of the Maass 
wave forms of Tq(N) . For the group To (4) and Selberg's char- 
acter Xa there exists just one non-trivial symmetry operator P. 
The eigenfunctions of the corresponding reduced transfer operator 
with eigenvalue A = ±1 are related to Maass forms even respec- 
tively odd under a corresponding automorphism. It then follows 
from a result of Sarnak and Phillips that the zeros of the Selberg 
function determined by the eigenvalues A = —1 of the reduced 
transfer operator stay on the critical line under the deformation of 
the character. From numerical results we expect that on the other 
hand all the zeros corresponding to the eigenvalue A = +1 leave 
this line for a turning away from zero. 
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1. Introduction 

In the transfer operator approach to Selberg's zeta function for a 
Fuchsian group T this function gets expressed in terms of the Fredholm 
determinant of this operator which is constructed from the symbolic 
dynamics of the geodesic flow on the corresponding surface of constant 
negative curvature. Even if this approach has been carried out up to 
now only for certain groups like the modular subgroups of finite index 
[2], [3], [4], or the Hecke triangle groups [16], [14], [15] it has lead for 
instance to new points of view on this function [22] or the theory of 
period functions [12]. Another application of this method is a precise 
numerical calculation of the Selberg zeta function [20], which seems 
to be impossible by other means at the moment. In this paper we 
discuss the transfer operator approach to Selberg's zeta function for 
Hecke congruence subgroups with character, of special interest being 
the behaviour of its zeros for To (4) under the singular deformation of 
Selberg's character [19]. 

As found numerically by M. Fraczek in [9], certain symmetries of the 
transfer operator for these groups play thereby an important role. 
These symmetries lead to a factorization of the Selberg zeta function 
as known for the full modular group SX(2, Z). There it corresponds to 
the involution Ju(z) = u(—z*) of the Maass forms u for this group [8], 

[12]. Obviously the corresponding element j = ^ G GL{2, Z) 

generates the normalizer group of SX(2,Z) in GL(2,Z). It tuns out 
that also the symmetries of the transfer operator for T (N) correspond 
to automorphisms of the Maass forms from its normalizer group in 
GL(2,Z). 

For the group r (4) with a character Xa introduced by Selberg in [19] 
and discussed also by Phillips and Sarnak in [18], there is only one 
such non-trivial symmetry of the transfer operator. It corresponds to 
the generator of Fo(4)'s normalizer group in GL(2, Z) leaving invariant 
the character Xa- Results of Sarnak and Phillips imply that the zeros 
on the critical line of one factor of Selberg's function stay on this line 
under the deformation of the character, and hence the corresponding 
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Maass wave forms for the trivial character remain Maass wave forms. 
Numerical results [9] on the other hand imply, that the zeros on the 
critical line of the second factor of this function should all leave this 
line when the deformation is turned on. A detailed discussion of these 
numerical results and their partial proofs is in preparation [1]. 
The paper is organized as follows: in Section 2 we recall briefly the 

form of the transfer operator L^ Pir = r _ ^ for a general finite 

\ L p,* u ) 

index subgroup T of the modular group SL(2, Z) and unitary represen- 
tation 7r and introduce the symmetries P = ^\ of this operator 

defined by permutation matrices P. Any such symmetry leads to a 
factorization of the Selberg zeta function in terms of the Fredholm de- 
terminants of the reduced transfer operator PC^ n . The eigenfunctions 
with eigenvalue A = ±1 of this reduced transfer operator then fulfill 
certain functional equations. In Section 3 we discuss the generators 
J n - of the group of automorphisms in GL(2,Z) of the Maass forms u 
for T = Tq(N) and n = xo the trivial character. We introduce their 
period functions ip and derive a formula for the period function J n -i\) 
of the Maass form J n -U. In Section 4 we introduce Selberg's charac- 
ter Xa and the non-trivial automorphism J 2j _ of the Maass forms for 
r (4). We derive again a formula for the period function Jz-x/j of the 
Maass form J 2) _u leading to a permutation matrix P 2j _ which defines 
a symmetry P 2 _ °f t ne transfer operator . From this we con- 

clude that the eigenfunctions with eigenvalue A = ±1 of the operator 
P 2 _>Cg j7r correspond to Maass forms even respectively odd under the 
involution J 2) _. Former results of Phillips and Sarnak then imply that 
the zero's of the Selberg function on the critical line corresponding to 
the eigenfunctions with eigenvalue A = — 1 of this operator stay on this 
line under the deformation of the character. 



2. The transfer operator and Selberg's zeta function for 
Hecke congruence subgroups To (A) 

The starting point of the transfer operator approach to Selberg's 
zeta function for a subgroup T of the modular group SL(2, Z) of index 
\x = [SL(2, Z) : T] < 00 is the geodesic flow $ t : SM T -> SM T on the 
unit tangent bundle SM-p of the corresponding surface Mp = T \ H of 
constant negative curvature. Here H = {z = x + iy : y > 0} denotes 
the hyperebolic plane with hyperbolic metric ds 2 = dx ^ 2 dy on which 

the group T acts via Mobius transformations gz = if g = 
In the present paper we are mostly working with the Hecke congruence 
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subgroup 

T (N) = {geSL(2,Z):g= ^ % 
with index /i^r = N T7 (1 + where p is a prime number. If p : T — >■ 

p|JV p 

end(C d ) is a unitary representation of T then Selberg's zeta function 
Zy, p is defined as 

oo 

(2.0.1) Zr iP (P) = J] II det (! - P(^) exp(-(A; + /3)/ 7 )) , 

where / 7 denotes the period of the prime periodic orbit 7 of $ 4 and 
<7 7 G T is hyperbolic with ^(7) = 7. In the dynamical approach to this 
function it gets expressed in terms of the so called transfer operator 
well known from D. Ruelle's thermodynamic formalism approach to 
dynamical systems. For general modular groups T with finite index p, 
and finite dimensional representation ir this operator Lg )7r : B — >■ B 
was determined in [2], [3] as 

( n r A 

(2.0.2) L Air 



A, 



where i? = 5(-D, O 1 ) 05(£), C M ) is the Banach space of holomorphic 
functions on the disc D = {z :\ z — 1 |< |}, and p n denotes the repre- 
sentation of SL(2, Z) induced from the representation tt of T whereas 

£t,P, is S iven for 3^/5 > I by 

00 1 

(2-0-3) {C% p J_) (z) = £ j—^pAST^m 



~ x (z + n) 2 $ - z + n 

where S = ^ and T — ( J J J . In the following we restrict 

ourselves to one dimensional unitary representations tt, hence unitary 
characters, which we denote as usual by x- m this case the following 
Theorem was proved in [3]. 

Theorem 2.0.1. The transfer operator Lg iX : B — »■ B with L^ )X = 
L° and (C% x f) (z) = jg I - T I_ FPx (^T ± «)/(^) extends to 

a meromorphic family of nuclear operators of order zero in the entire 
complex (3 plane with possible poles at (3^ = k = 0, 1, 2, . . .. The 
Selberg zeta function Z^ tX for the modular group T and character \ 
can be expressed as Z rx (/3) = det(l — Lg jX ) = det(l — C^ X C^ X ) = 
det(l-£^£+ x ). 

This shows that the zero's of Selberg's function are given by those 
/3-values for which A = 1 belongs to the spectrum <t(L^ jX ) respectively 
a{C^ x Cp x ) = a(Cp tX Cp x ). From Selberg's trace formula one knows 
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that there are two kinds of such zeros: the trivial zeros at 3 — — /c, k — 
1,2,..., and the so called spectral zeros. They correspond either to 
eigenvalues A = 8(1 — 8) of the automorphic Laplacian with 1R.B = | 
or \ < 8 < 1 respectively to resonances of the Laplacian, that means 
poles of the scattering determinant with dt8 < \ and Q8 > [21] [10]. 
For arithmetic groups like the congruence subgroups with trivial or 
congruent character \ one knows that these resonances are on the line 
dt/3 = |, corresponding to the nontrivial zeros (r{28) = of Riemann's 
zeta function Cr when assuming his hypothesis, respectively on the line 
3?/3 = 0. For general Fuchsian groups and congruence subgroups with 
non-congruent character however these resonances can be anywhere in 
the halfplane < \. 

2.1. Symmetries of the transfer operator for Tq(N). It turns 
out that there exists for any N a finite number of x permu- 
tation matrices P with P 2 = id^ N such that the matrix P = 
commutes with the transfer operator L^ jX and hence 
(2.1.1) PCl x = Cl x P 

Thereby P = (Pij)i<i,j<^ N acts in the Banach space B(D 1 <C I1N ) as 

A* AT 

(Pf)i(z) = £ Pijfj(z) if f(z) = {fi(z))i<i< m . We call such a matrix 
_ ~~ j=1 

I ' a symmetry of the transfer operator. As an example consider the 
group To (4) and Selberg's character Xa,0 < ol < 1, which will be 
described later. Its transfer operator L / 3 jXq has the following form 

La, Xa .f+i 



P3 



= E f-3\2pST 1+4 « + f.,\ w ~ST 2+4q + f^ST 3+4q 

q=0 

+ f. 2 \ 2p ST 4+4q 

oo 

= e 2ni(l+4q)a f_ 1 \ 2/3 §T 1+4q + e 2ni ( 2+4< l) a f_ 1 \ 2 / 3 ST 2+4q 

q=0 

_|_ e 27ri(3+4g)a \ 2j3 ST 3+4q + e 27Ti ( i+i l) a /„ : 1 2/3 ST 4+4<? 

oo 

= J2 e~ 2nia f- 2 \ w ST l+4q + e- 2ma f- 3 \ 2 pST 2+4q 

q=0 

+ e- 2nia f- 4 \ 2 pST 3+4q + e- 2nia f- 5 \ 2 pST 4+4q 

oo 

= J2 e~ 27ria{1+4q) f- 6 \ 2 pST 1+4q + e~ 2ma{2+4q) f. & \ 2 pST 2+4q 

q=0 

+ e" 2 ™ {3+4<?) f^ & \ w ST 3+4q + e~ 2nia{4+4q) f. 6 \ 2 pST 4+4q 

oo 

e J-A\2fiOl H +e J- 5 \ 2 /3bl 

q=0 

+ e 2ma f_ 2 \ w ST 3+4q + e 2nia f^ 3 \ 2 sST 4+4q 
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oo 

W/+6 = E/-5l2^T 1+4 ^ + /_ 2 | 2/ 35T 2+ ^ + /_3| 2 ^T 3+4 " 

+ f-^pST^ 

oo 

= E/+5| 2 ^r 1+ ^ + / +4 | 2 ^T 2+ ^ + / + 3| 2 ^T 3+4 " 

+ / +2 | 2/3 ST 4+4 « 



oo 



,/_ 2 = E e " 2 ™ (1+4<?) /+i|2^T 1+4<? + e- 2 ™ (2+49) /+i|2^T 2+4,? 



9=0 



+ e- 2 ™ (3+4,?) /+l|2/3^T 3+49 + e -2™(4+4 g ) /+i | 2 ^ T 4+4 (? 

oo 

IW-3 = E e " 2 ™/+4|2/35T 1+49 + e- 2 ™/ + 3| 2 ^T 2+4<? 

g=0 

+ e- 2 ™/ +2 | 2 ^T 3+4g + e-^U^ST 4 ^ 



,/_ 4 = E e2 " a(1+49) /+6|2^T 1+49 + e 2 ™ (2+49) /+6|2^T 2+49 



9=0 



+ e 2 ™ (3+49) /+6| 2 ^T 3+4g + e 2 ™ (4+4,?) /+6|2^T 4+4,? 

oo 

W/-5 = E e2ma /4-2|2^T 1+4 « + e 2 ™/ +5 | 2 ^T 2+4 ^ 

9 =0 

+ e 2 ™/ +4 | 2/3 5T 3+49 + e 2lHa f^\ 20 ST A+Aq 

oo 

W/-6 = E/+3|2^T 1+4 « + / +2 | 2 ^T 2+ ^ + / +5 | 2 ^r 3+4 " 

9=0 

where / G B(D, C) B(D, C) is given by / = (/ + , /_) and / ± = 

(/±i)i<j<6 an( i ^ = ~- The induced representation p x of the character 
X on r (4) is defined in terms of the coset decomposition of SL(2, Z) 

(2.1.2) SL(2,Z) = \jT {4)R i 

i=i 

as 

(2.1.3) p x {g) ij = Sr^RigRj^xi^gRj 1 ), 1 < z, j < 6. 

Thereby we have chosen the following representatives i?j G SL(2, Z) of 
the cosets T (4) i?j 

(2.1.4) i?! = id 2 , Ri = ST~ 2 , 2 < i < 5 and i? 6 = ST 2 S. 

It turns out that the two permutation matrices Pi, i = 1,2 correspond- 
ing to the permutations a, with 

,„ . ^ 1 2 3 4 5 6 
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(0 . 1 2 3 4 5 6 

(2 - L6) ff2 = 6 4 3 2 5 1 

fulfill equation (2.1.1) for a = and hence the corresponding matrices 
Pi, i — 1, 2 commute with the transfer operator L^ xo where Xo is the 
trivial character. The matrix P 2 on the other hand commutes even 
with the operator L /3iXq for all a. Indeed the matrix p X0 (S) is given by 
the permutation as where 

/o 1 ^ 1 2 3 4 5 6 

(2 ' L7) as= 2 1 5 6 3 4 

and an easy calculation shows that Pip X0 (S) = p X0 (S)Pi, i = 1, 2. The 
matrix p X0 {T) on the other hand is given by the permutation cr^ with 

(2.1.8) ffr 1 2 3 4 5 6 



1 3 4 5 2 6 



One then checks that pp Xo (T) = p Xo (T ^p, i = 1, 2. Therefore 
PiP X0 {ST n ) = p Xo {ST- n )Pi for all n e N and 2 = 1, 2. For the 
character analogous relations hold for P 2 . 

For the trivial character xo one can determine for the group Tq(N) the 
number of matrices P, with the above properties and hence defining 
symmetries of the transfer operator as follows: 

Theorem 2.1.1. For the Heche congruence subgroup T (N) and trivial 

character Xo = 1 there exist matrices P = commuting 

with the transfer operator ~Lp, Xo where P is a pn x Pn permutation 
matrix with P 2 = 1 MJV and Pp Xo (S) = p Xo (S)P respectively Pp Xo (T) = 
p X0 (T~ 1 )P and hence 

p r+ — r- p 

Thereby = max{k : k | 24 and k 2 \ N}. The permutation 
matrices P are determined by the generators j of the normalizer 
group A/jv ofTo(N) in GL(2,Z). The Selberg zeta function Zr, Xo can 
be written as 

Z r , X0 = det(l - PCIJ det(l +PCIJ. 



Remark 2.1.2. For To (4) obviously hjsr = 2 and there exist according 
to Theorem 2.1.1 two such permutation matrices Pi and P 2 which in- 
deed are given by the aforementioned permutations <7j, i = 1,2. Since 
P 1 P 2 = P 2 P x and Pi £+ xo = £p M P u % = 1, 2 we find 

Pi P 2 Pi £j X0 = Pi P 2 £^ >Xo Pi = Pi C^ X0 P 2 Pi = Pi Cp, Xo Pi P2 
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and the operators Pi P 2 and Pi £^ iX0 commute, where the operator 
Pi P 2 corresponds to the permutation 

(2 19) a= 1 2 3 4 5 6 

l^-yj ° 6 4 5 2 3 1' 

We find also P 1 P 2 C + Pxo = C^ XQ P 1 P 2 . But (P X P 2 ) 2 = id 6 , hence 
this operator has only the eigenvalues A = ±1 and the Banach space 
P(P>,C 6 ) decomposes as P(P>, C 6 ) = B(D, C 6 ) + © B(D, C 6 )_ with 
P x p 2 / ± = ±/ ± for / ± G B(D, C 6 ) ± . The elements £ G B(D, C 6 ) e , e = 

± have therefore the form (f_)i — fi, 1 < % < 3 respectively (f ) a (i) = 
e/i, 1 < i < 3. Denote by 

£+ X0i± :P(P,C 6 ) ± ^P(A' 

respectively 

Pi£+ XOj± :P(P,C 6 ) ± ^P(P, 

the restriction of the operators £^ xo respectively Pi £^ xo to the sub- 
space B(D, C 6 )±, which obiously is isomorphic to the space B(D, C 3 ). 
Then det(l ± Pi C% Xo ) = det(l ± P £ft Xo ,+) det(l ± Pi £j X0) _), where 
the operator Pi £g e : P(P, C 3 ) ->■ P(P, C 3 ) can be written as 

£^ 

Cp t i + ^£^3 e£^,2 + £/3,4, 

oo 4 

with £ ftfc / = £ f\2,sST 1+k i, i < A; < 4 and £ = E £ The 

g=0 fc=l 

operator £^ Xo e in the space B(D, C 3 ) on the other hand has the form 



(2.1.11) £ 



( e£/3,2 + £3,4 + £/3,3 



/3,X0 e 



£p 

\ e£/3,i + £3,3 £p,2 + e ^,4y 

To relate the Fredholm determinants of the operators (Pi£t J 2 and 
(£p e ) 2 we use the following simple Lemma 

Lemma 2.1.3. Let be a, (3 and'j complex numbers and e = ±1. Then X 

/0 a (3\ 

is an eigenvalue of the matrix L x = 7 \ iff eA is an eigenvalue 

\0 (3 eaj 

/0 a 

of the matrix L 2 = 7 

e(3 a / 

Proof. The proof follows from the characteristic polynomial of the 
two matrices. □ 

This shows that trace L? = £ (LJ) fcfc = e" trace L™ = e n J2 {^ 2 ) kk 

k=l ' k=l 

for all n G N. But then is not too difficult to see that also trace (£g Xo e ) n 
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e" trace (Pi £^ xo J n for all n e N and hence det(l - (Pi£j w ) 2 ) = 
det(l — (£~p iXoe ) 2 ) for e = ±. Therefore the Selberg zeta function 
Zr (4),xo{P) f° r ^e group r (4) with trivial character xo can be writ- 
ten as 

ZrodUoW) = det (l - (Pi£j )X0 ) 2 ) = det (l - (£+ xo ) 2 ) 

(2.1.12) = det(l-£+ Xo )det(l + £+ Xo ) 
Furthermore this function factorizes in this case also as 

Zr oi ,), x M = det(l-P 1 £+ XOi+ )det(l-P 1 £+ XOj _) 

(2.1.13) x det(l + P 1 jC+ XOt+ )det(l + P 1 jC+ xo j 

To prove Theorem 2.1.1 we relate the matrices P to the generating 
automorphisms in GL(2, Z) of the Maass wave forms for Tq(N) and 
can determine this way the explicit form of these matrices P. For 
this we derive in a first step a Lewis type functional equation for the 
eigenfunctions of the operator P C^ x with eigenvalue A = ±1. 

2.2. A Lewis type functional equation. Consider any finite 
index modular subgroup T and any unitary character x '■ T — > C* 
respectively the induced representation p x of SL(2,Z). Assume there 

exists a symmetry P = with P a permutation matrix with 

the properties analogous to Theorem 2.1.1, and commuting with the 

/ C + \ 

transfer operator L^ x = ^ Px of T. If / is an eigenfunction 

of the operator P C^ x with eigenvalue A = ±1 then one can show 

Proposition 2.2.1. I/P£j x /(C) = A/(C) with A = ±1 then the 
function \j/(C) := F 'p x (T~ l S)P f '(( — 1) fulfills the functional equations 

(2-2.1) 3L(0 = *C 2 "Pp x (S)M^ 

respectively 

(2.2.2) *(C) " P X (T^) MC + 1) " (C + 1)" 2/3 P X {T'- 1 ) M^y) = 0, 

where T' = ST~ 1 S. On the other hand every solution of equa- 
tions (2.2.1) and (2.2.2) holomorphic in the cut [3-plane (— oo, 0] with 
^i(z) = (£-»"Mi,2Rs}) as z 10, respectively %(z) = ofr™^^ 1 *) 
as z — > oo ; determines an eigenfunction f with eigenvalue A = ±1 of 
the operator P £~p x . 

Proof. Let > \. If P£j/(C) = A/(C), A = ±1 then obviously 
P Px (STS)PP£^l(( + l) = \Pp x (STS)Pf((+l). Subtracting the two 
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equations leads to 

A/(C) - \P Px (STS)Pl(( + 1) - (C + l)-^Pp x (5T)/(^l I ) = 0, 
and hence the function '■— Pf(C ~ 1) fulfills the equation 

(2.2.3) ±(0 - Px (STS)±(( + 1) - XC 2 ^p x (ST)Pl(^) = 0. 

Replacing there C by ^ and multiplying the resulting equation by 
C^p x (STS)P Px (T^S) gives 

C^ Px {STS)P Px {T^S)±(^) - C w Px {STS)P Px {S)±{^)- 

-\ Px (STS)±(( + 1) = 0. 
Since p x (S)P = Pp x (S) one finds, comparing with equation (2.2.3), 

t(0 = XC W P x (STS)Pp x (T~ 1 S)t(^). 

Hence the function t(j := p x (T~ l S)ip fulfills equation (2.2.1). The same 
equation is then fulfilled also by the function 

(2.2.4) *(C) := P±(C) = PPx(T~ l S) P f{( - 1), 
that is 

(2.2.5) *(C) = XC W Pp x (S)m^)- 

Inserting finally t^(C) = P x (ST)P}5/_(0 into equation (2.2.3) and using 
(2.2.1) leads to the equation 

mo - pp x (T)pmc + 1) - (c + iy 2p p Px {T')PM^) = q. 

But by assumption Pp x {T)P = p x (T~ 1 ), hence Pp x (T')P = p x (T /_1 ) 
and therefore 

(2.2.6) mo - pAt-'mc + 1) - (c + ir 2 "p x (r^m(^) = q. 

Hence for > | the first part of the proposition holds. By analytic 
continuation in f3 one proves the general case. 

To prove the second part we follow the arguments of Deitmar and 
Hilgert in [7] (see their Lemma 4.1): if 2(C) is a solution of the Lewis 
equation (2.2.2) with f3 ^ Z then 2 has the following asymptotic ex- 
pansions: 

1 oo 

mO ~c-o C 2/3 Qo(^) + E CK l 



l=~l 



mO -c^oo Qco(C) + E cfC 



oo 

1-2/3 



l=-l 
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where Qo,Qoo ■ C — > C M are smooth functions with Qo(C + 1) — 
p x (T')Qo(C) respectively Qoo(C + 1) = PxCOQoo(C) and the constants 
C_i and C^' are determined by the Taylor coefficients C_ m = -^2 (1)- 
The functions Qo and Qoo are defined as follows for general /3 with 
< M G N: 

i M 

Qo(C) := r 2/3 ^( 7 )- EC*(™ + 2)MG» 

S m=0 

oo / i M N 

- E(^+cr 2/3 Px(^T- i )(£(i+-^ : )-5: 

n=0 

respectively 

m=0 

E(n+ Q-WpxiT-^r-*) 2(1 - - E 



Qoo(c) := mo- EC(m+2/3,c+i)c:„ 



n=0 \ ' m=0 



Thereby 



and 



1 N ~ 1 4- C 

Cp x KC) = ^ r EPx(^T- 1 )C(a,^) 
C; x KC) = ^EPx(T- fc T'- 1 ) C H(a,^) 

fc=0 

with Ci?( a jC) the Hurwitz zeta function. According to Remark 4.2 in 
([7]) any solution 2 of equation (2.2.2) with 2(C) = o(C~ min{1,2/3} ) for 
C — > fulfills the equation 

OO 1 

2(c) = c w E(« + c-^-^px^'-^- 1 )^! + -— — ) 

n=0 " "r S> 

and moreover = 0. But if 2(C) fulfills also the equation (2.2.1) 
then one finds 

1 00 1 

\C 2 "P Px (sm 7 ) = C 2p £(" + O-^Cr^r-^i + — -) 

S n=0 ^ 



and hence 

n + C 



(2.2.7) AP Px (5)2(C+l) = E("+0" 2 V(r / - (n - 1) T- 1 )2(l+ 



According to equation (2.2.4) 2(C + 1) = p P x ( T ~ 1S ) P K0 and hence 
we get 

OO 1 

Ap x (5T- 1 5')P/(C) = Y.( n +0- 2 "p x (T'- {n - 1) T~ 1 )P Px (T- 1 S)Pl(-—). 

n=l S + n 
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Inserting J^M™- 1 ) = ST^ n ~^S one arrives at 

OO 1 

A/(0 = E( n + 0- 2 "Pp x (ST n )p x (ST- 1 )P Px (T^S)Pl(——). 

n=l S "r " 

Since p x (ST~ 1 )P = Pp x (ST) we get finally 

Hence any solution \j/ of the Lewis equations (2.2.1) and (2.2.2) with 
the asymptotics at the cut ( = determines an eigenfunction / of the 
transfer operator PC^ X with eigenvalue A = ±1. 

□ 



3. Automorphism of the Maass forms and their period 

functions for T (N) 

The Maassforms u = u(z) of a cofinite Fuchsian group T and unitary 
character \ are rea l analytic functions u : H — > C with 

• A u(z) = X u(z), 

• u{gz) = x(g) u(z) for all g G V, 

• u(gjz) = 0(y c as y — > oo for some constant C G K and all 
cusps Zj = gj(ioo) of V. 

The cusp forms are those forms which decay exponentially fast at the 
cusps. If u G L 2 (Mr) we call u a Maass wave form. 

Definition 3.0.1. An element j G GL(2, Z) defines an automorphism 
J of the Maass wave form u for the group V and character % if Jm with 
Ju(z) := is a Maass form for T and character \. 

Obviously j defines an automorphism J iff j is a normalizer of the 
group T and the character \ is invariant under j, that is x{j 9 j 1 ) = 
x(g) for all g G T. Thereby = ^*f^ if det g = ad—bd — — 1. We have 
to show that the function Ju(z) = u(jz) has at most polynomial growth 
at the cusps Z{ = r»(ioo) of T, where 73 G SL(2,Z) . If det j = — 1, 

one has u(jTi(z)) = u(jnj -jo,-(z)) where j - = • Then 

j T ijo - ^ SL(2, Z) and hence jTij _ = ^iRi for some 7« G T and some 
representative Ri of the cosets T \ S'L(2, Z). But i?,, = r/T^j) for some 

?7 G T and some index cr(«) . Hence u{jTi{z)) = u {r a ^{—z*)^ which is 
at most of polynomial growth at the cusps. The same argument applies 
if det j = l. It shows also that Ju is a Maass wave form or a cusp form 
if u is one. 
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3.1. The group of automorphisms of Maass forms for T (N) 
and trivial character xo- We restrict ourselves now to the case 
T = r (iV) and assume X = Xo- Denote by A/jv the normalizer 
group {r (A r )j : j normalizer of T (N) in GL(2,Z)}. Using results 
by Lehner and Newman [11] respectively Conway and Norton [6] we 
find 

Proposition 3.1.1. For = max{r : r | 24 and r 2 | N} and k^ : = 

the normalizer group Af^is given by A/jv = {r (A r ) j n ,±, j n ,± = 

(nl N ± ° 1 ).0<«<^-1} 

Proof. Using the fact that the divisors k of 24 are exactly the 
numbers for which a ■ d = 1 mod k implies a = d mod k one shows 
that the normalizer group of T (N) in SX(2,Z) is T (N) \ T (%) [11] 
with v = 2 min ^ 3 '[f 1> • 3min{i,[f ]} and e2 = max {/ : 2 l \ N} respec- 
tively e 3 = max{Z : 3' | N}. But obviously v = h N and [r (/cAr) : 
r (iV)] = /ijy and hence A/jv = r (iV) \ (r (k N ){jr (k N )j ,-). Since 
Jn,± 7^ jm,± mod r (A r ) for n ^ m, this group has just the 2/ijv ele- 
ments T (N)j nt ±, < n < hpf — 1. The normalizer group A/jv is there- 
fore generated by the hN generators {T (N)j n ^, < n < h^ — 1}. □ 

3.2. The period functions of r (A r ) and character x- For u a 

Maass form with Au = (3(l-fi)u and T (N)\SL(2, Z) = {r (A r )i? i , 1 < 
z < /^Af} its vector valued period function u is defined by 

(3.2.1) u = (Wi(^))i<i< w where «;(z) = u^z) 
Then one has as shown for instance in [17]: 

• U.(gz) = Px(g)u.( z ) for all g G SX(2, Z) and p x the representa- 
tion of SL(2, Z) induced from the character x on ro(iV) 

• A«i(z) = 0(1 - 1 < z < Atjv- 

Given next two eigenfunctions u = u(z) and u = 17(2) of the hyperbolic 
Laplacian with identical eigenvalue A = /3(1 — (3), one knows [12] that 
the 1- form 77 = 77(14, v) with 

r](u,v)(z) := v(z)d y u(z) — u(z)d y v(z)dx + [u(z)d x v(z) — v(z)d x u(z)]dy 

is closed. If u = u(z) is a Maass wave form for T (N) with eigenvalue 
A = 0(1 — (3) and R^(z) = ^_ x y +y 2y denotes the Poisson kernel, the 
vector valued period function = (V'j ( C) ) i<? </xjv i s defined as 

00 

(3.2.2) Vi(0== fv(uj,R%)(z). 



The following result has been shown for trivial character xo by Miih- 
lenbruch in [17]. His proof can be extended however immediately to 
the case of a nontrivial character x- 
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Proposition 3.2.1. The period function ip — V'(C) °f a Maass wave 
form u = u(z) for Tq(N) and unitary character \ is holomorphic in the 
cut (-plane C\ (—00, 0] and fullfills there the Lewis functional equation 
(2.2.2) 

t(o - PxiT-'mc + 1) - (c + ir 2 ^'- 1 )^^-) = 0, 

where p x denotes the representation of SL(2,Z) induced from the char- 
acter x of t (n). 

On the other hand it follows from the work of Deitmar and Hilgert 
in [7] that the solutions of the above equation holomorphic in the cut 
C-plane with certain asymptotic behaviour at the cut and at 00 are 
in one-to-one correspondence with the Maass wave forms. Their paper 
treats only the trivial character but it can be extended also to the case 
of nontrivial character \. Since the function }&_(() = Pp x (T~ 1 S)Pf(( > — 
1) with / an eigenfunction of the operator PC^ with eigenvalue A = 
±1 is such a solution of equation (2.2.2), these eigenfunctions are in 
one-to-one correspondence with the Maass wave forms. As in the case 
of the full modular group SL(2, Z) treated in [5] respectively in [12] 
one can extend this result to arbitrary Maass forms, that is also to the 
real analytic Eisenstein series for r (A) and unitary character x 

3.3. Automorphisms of the period functions. We have seen 
that the group of automorphisms in GL(2, Z) of the Maass forms u 
of r (A) and trivial character xo is generated by the matrices j n _ = 

^^j ,0 < n < /In — 1. Denote by J n -U the Maass form 

J n -u(z) := u{j n -z) and by J n -^ its period function. Then one shows 

Theorem 3.3.1. The period function J n -ip = {J n -ipj{C,))i< i j< l j, N is 
given by 

(3-3.1) Jn,-M0 = C 2/ Va„_<w (j )(^), 

where the permutations A n _,<r and 5 are determined through the coset 
representatives Rj ofT (N) \ SX(2,Z) as follows: 

jn,+Rj = 9jR\n,-(j)i jo,-Rjjo - = 7jR<r(j) andRjS = r}jRs(j) 
with 9j,jj, r}j E T (N) for I < j < fi N 

Proof. For u = u(z) a Maass form for To(A) and trivial character 
Xo and u = u(z) its vector valued Maass form consider the Maass forms 
J n ,±u{z) = u(j nt ±z) respectively J n ,±u(z) = (Jn,±v>j(z))i<j<n N with 
J n ±Uj(z) = u(j nt ±Rjz). Since j n ,+Rj = 9jR\ n .-{j) f° r some uniquely 
defined 9j G T (A) and permutation A ni _ of {1, 2, ... , [i N } one gets for 

(3.3.2) J^+ujiz) = u(R Xn _ {j) z) = u Xni _{j)(z). 
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For J n!+ Uj(— z*) = u(j nj+ Rj(—z*) = u(j nt+ Rjj -z) on the other hand 
one finds 

J n>+ Uj(-z*) = u(j n „j -Rjjo,-z) = u(j n _R a(j) z) 

since jo-Rjjo,- = JjRa(j) f° r some unique jj G T (N) and permutation 
a of {1, 2, ... , /iTv}. Hence 

(3.3.3) J nt+ Uj(-z*) = J n -u a{ j){z). 

Consider next J nj+ Uj(Sz) = J nt+ u(RjSz). Since RjS = r)jRs(j\ for 
unique r]j G T (A r ) and permutation S of {1, 2, ... , ^n}, one has 

J ni+ Uj(Sz) = J nt+ u{R S fj)z) = J nt+ u 5 (j){z). 

Hence by equation (3.3.2) 

(3.3.4) J n;+ Uj(Sz) = u Xn> _o6(j){z)- 

On the other hand one gets for J nt+ Uj(S(— z*)) = J nj+ Uj(—Sz*) by 
using equation (3.3.3): 

J nj+ Uj(S(-z*)) = J n _u a{j )(Sz) =u(j n _R a ( j )Sz), 

and therefore 

J n>+ Uj(Sz*) = u(j n -rj a ^)R S oa(j)(z)) = J n -U S oa(j)(z). 

But a o 8 = 8 o a and therefore 

(3.3.5) J n>+ Uj{S{-Z*)) = Jn,-U a0 5(j)(z). 

Define next 

v±j(z) := J ni+ Uj(z) ± J ni+ Uj(-z*). 
Then by equations (3.3.2) and (3.3.3) one has 

V±j(z) = U X (j)(z) ± J n -U a (j){z) 

and hence, if Au(z) = (3(1 — (3)u(z), 

(3.3.6) Av ±J (z)=P(l-P)v±j(z), 
respectively 

(3.3.7) v ±J (-z*) = ±v ±d (z) 
Equations (3.3.4) and (3.3.5) on the other hand show 

(3.3.8) v ±J (Sz) = v± m (z). 

ioo 

Set "01(C) := / v(v±i Rf)( z )- Then, since v± ,•(— z*) = ±v±j(z) one 
o 

finds [12] 

(3-3-9) ^AO = Wf { ^P +1 dt, 
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respectively 



OO 

(3.3.10) ^(0 = -/ 

o 




Using next the identity (4.0.23) one easily shows 



(3-3.11) ^0 = ±C 2 ^W7)- 




Therefore 



(3.3.12) 



Remark 3.3.2. As can be seen from their action on the coset repre- 
sentatives Rj the permutation 5 commutes with the permutations \ n - 
and a. Furthermore one has a 2 = S 2 = (X n ,- ° o~) 2 = id where id 
denotes the identy permutation. This shows also that the automor- 
phisms J n - are involutions both of the Maass forms and the period 
functions, a special case of these involutions for all groups T (N) being 
Jq-u(z) = u(-z*). 

Denote by Q n .-, < n < /ij\r — 1, the p^ x Pn permutation matrix 
corresponding to the permutation A n> _ o a o S. Then the following 
Theorem holds: 

Theorem 3.3.3. The permutation matrices P n - '■= Pxo(S)Qn,-, < 
n < hjsi — 1, define symmetries P n - = \ n | f or the trans- 



Xo = 1 with P%_ — id^ N and P n _p Xo (S) = p X0 (S)P n - respectively 



P n _p Xa {T) = p Xo (T l )P n - and therefore P n _£^ Xo = C^ XQ P n -. The 



permutation matrix P n - is determined by the permutation A ni _ o a and 
hence by the coset representatives j n _Rjj ^. 




□ 




character 
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Proof. Since the matrix P nj _p X0 (S) is determined by the coset 
representatives j n -RjSj - whereas p X0 (S)P n _ is determined by the 
coset representatives j n _Rjj _S and jo-S = Sjo- we get P n _p Xo {S) = 
p X0 (S)P n -. On the other hand Tjo- = jo-T^ 1 and therefore P„„p X0 (T) 
p Xo {T~ l )P n _ and hence the Theorem is proven. □ 

Obviously Theorem 2.1.1 follows now from Theorem 3.3.3. For 
the automorphisms j n>+ = j n -jo- one gets the symmetry P ni+ = 

r^Q + ^ with P n>+ the permutation matrix corresponding to the 

permutation A Wj _ oa o \ 0> _ o a determined by the coset representatives 

jn,+Rj- 

Remark 3.3.4. The symmetry Pq,- is given by p Xo (SM) where M = 

K ^ and p xo denotes the representation of GL(2, Z) induced from 

the trivial character xo of Fq(N). The transfer operator C^ M of Manin 
and Marcolli for T (N) introduced in [13] turns out to coincide with 
the operator p X0 (S , )P -£p p X0 (S) and appears as a special case of our 
operators P n -C^ XQ . 

Corollary 3.3.5. The permutation matrices P n ,-, < n < hpf—l, gen- 
erate a finite group consisting of the permutation matrices {P n ,±, < 
n < /ijv — 1} and isomorphic to the normalizer group 
mathcal(N)N ofTo(N) in GX(2, Z). The symmetries {P n ,±, <n < 
— 1} of the transfer operator L/3 iXo for Tq(N) and trivial character 
Xo define a finite group isomorphic to the group Mn- 



4. Selberg's character Xa for r (4) 
The group r (4) is freely generated by the two elements T - 
and B — ( \ °Y Hence any g G r (4) can be written as g 



1 1 
1 



fl T mi B ni . If Cl(g) = mi then Selberg's character Xa [19] is defined 

i=i i=i 
as 

(4.0.15) Xa(g) = exp(27nafl(g)), < a < 1. 

Denote by Zi, 1 < i < 3 the inequivalent cusps of r (4) and by the 
generators of their stabilizer groups T Zi with TiZi = Zi. They can be 
taken as Z\ = ioo, Z2 = 0, z% = — \ and T\ = T,T 2 = B, T 3 = T^B^ 1 . 
The character Xa is singular in the cusp iff Xa(Ti) = 1- Otherwise the 
character is non-singular in Z\. It is well known that the multiplicity 
K {Xa) of the continuous spectrum of the automorphic Laplacian A 
with character Xa is given by k(xo) = : Xa(Ti) = 1}- Therefore 
K (Xa) = 3 for a = whereas k(xo) = 1 for a ^ and hence the 
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multiplicity of the continuos spectrum of the Laplacian changes from 
3 to 1 when the trivial character is deformed to Xa with a / 0. It 
is known [18] that the character Xa is congruent (or arithmetic) iff 
a G {&|, < k < 4}. Since the Selberg zeta function given in (2.0.1) 
has the property Zr (4), Xa = Zr (4),x- Q an d obviously Xa = Xa+i we can 
restrict the deformation parameter a to the range < a < \. 

Lemma 4.0.1. The Selberg character Xa is invariant under the map 
j 2 -Z = 2z*-i an d Ji,- U { z ) := u {j2- z ) is a Maass form for T (4) and 
character Xa if u = u { z ) is such a Maass form. 

Proof. We only have to show that Xa is invariant under the map 
j 2 -Z = 2z*-i ■ ^ or 9 = T we find j\JTj 2 - = TB and hence 

Xa{h- T h-) = Xa(TB) = Xa{T), 

whereas for g = B one finds j 2 -Bj 2 - = B~ l and hence 

X«U2,-Bj 2 ,-)=Xa{B- 1 ) = x a (B). 

Therefore Xa{j% -932-) = xM for all g G T (4). □ 

For u = u(z) a Maass form for T (4) and character Xa an d ip = 
(V'i(C))i<j<6 its period function denote by J_w the Maass form J_u(z) := 
u{]2-z) respectively by = (J-^j(C))i<j<6 its period function. 

Then one shows 

Theorem 4.0.2. The period function J_tp of the Maass form J^u is 
given by 

(4.0.16) J-^j{C) = C 2P Xa{r]aoS(j))^\ 2 ^oao5(j){^) 

where the permutations \ 2 ~, o~, 5 respectively the rjj G T (4) are deter- 
mined through the coset representatives Rj by 

32,+Rj = 6jR\2-(j), 3o-Rjjo- = ljR<r{j), RjS = VjRs(j) 
with 8j, 7j, rjj G T (4) for 1 < j < 6. 

PROOF. Set j± := j 2; ± and J±u(z) : = u(j±z). Then J_u is a 
Maass form for To (4) and character Xa whereas J + u is a Maass form 
for To (4) and character X-a- The vector valued Maass form J + u = 
(J+Uj)i<j<6 is given by J + Uj(z) = u(j + Rjz). Therby we have choosen 

the representatives Rj of the cosets in SX(2, Z) = \J r (4)i?j as 

i<i<6 

follows: 

R 1 = id 2 , Rj = ST j ~ 2 , 2 < j < 5, Rq = ST 2 S. 
But j+Rj = 9jR\ 2 _(j) for some 9j G T (4) and some permutation A 2 _ 
of the set {1, 2, . . . , 6} and hence J + Uj(z) = Xa(Qj)u(R\ 3 (j)z). It turns 
out that 6j = B^ 1 for 1 < j < 3 and Q 3 - = id 2 for 4 < j < 6. Hence 
Xa{0j) = 1 and 

(4.0.17) J +Uj (z) = u X2i _ {j) (z), 1 < j < 6, 
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with \ 2j - the permutation 

,. n10 > N 1 2 3 4 5 6 

(4.0.18) A 2 ,_ = 6 4 5 2 3 x . 

Consider next J + Uj(—z*) = J + Uj(jo-z). Then 

J + Uj(jo-z) = utf+Rjjo-z) = u(j + j -jo-Rjjo-z). 

If jo-Rjjo- = IjRa(j) then J + Uj{j Q -z) = u(j^jj_j_R a ^)z). But it 
turns out that j-^jj- = id 2 for j = 1, 2, 6 respectively j-ljj- — B for 
j = 3,4,5, hence Xa(j-ljj- = 1 and therefore 

(4.0.19) J + uj(-z*) = J + Uj(j ,-z) = J-U a {j){z). 

Since furthermore J + Uj(Sz) = u(j + RjSz) = u(j + T]jRs(j)z) one finds 

(4.0.20) J + Uj(Sz) = X-a(Vj) u X 2 ,-o8(j)(z) 

where 5 is the following permutation of the set {1, 2, . . . , 6} 

,Ann-i\ r 1 2 3 4 5 6 

(4.0.21) ^= 2 1 5 6 3 4 - 

and r)j = id>2 for j = 1,2,4,6 respectively 773 = r/^ 1 = T~ X B~ X . 
For J + Uj(-Sz*) one gets with (4.0.19) J + Uj(-Sz*) = J_u a (j)(Sz) = 
u(j2-R a (j)Sz) and hence 

J + Uj(-Sz*) = u(j 2 -r] a (j)Rsoa(j)z) = Xa(v<r(j))J- u 6oa(j)(z). Using the 
explicit form of the rjj one shows Xa(Va(j)) = X-a(Vj) an d therefore 

(4.0.22) J + Uj(Sz*) = X-a(Vj)J-V>6oa(j)(z). 

Define next v±j = v±j(z) as 

(4.0.23) v±j(z) := J+u^z) ± J +Uj (-z*). 

Then v± tj {-z*) = ±v± d {z) and by (4.0.20) respectively (4.0.22) 

(4.0.24) v± tj (Sz) = X-a{r]j)v ±m {z) 

ioo 

If therefore V4?(0 := / v( v ± j> Rc)( z ) one S e t s from relation (4.0.24) 



(4.0.25) V4j(C) = ±C 2f3 X-«(VjW± m (^) 

and using the identity (4.0.23) 
(4.0.26) 

Adding these two equations leads finally to 

(4.0.27) J-4>j(0 = C^XaiVaoSU^X^oaoSU)^)- 

□ 
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Inserting the explicit form of the permutations 

1 2 3 4 5 6 



(4.0.28) ao 5 

respectively 
(4.0.29) 

and the character values 



2 1 3 6 5 4 



1 2 3 4 5 6 
A2 -° ao5= 4 6 5 1 3 2 



Xa{Vl) = XafaO = XaM = Xa(%) = 1 



respectively 
one finds 



XaiVs) = Xa(Vb) 1 



-2-nia 



(4.0.30) 



j-MO = C^Mt) 
J-MO ^ 
J-MO 



C^M^) 

A-2/3 -2ma 



J-MO 

J-MO 



-2/3 2-rria 



c ip m^ 



Define the matrix Q 2 - through the equation J 2i _^(C) = C 2 ^Q2,-' l l ) {\)- 
Then one gets 



Proposition 4.0.3. The permutation matrix P% := p Xa (S)Q2~ de- 

( P 2,^ 
\P2,- 



fines a symmetry P 2 



of the transfer operator 



J P,Xa 







for r (4) and characterxa vrithP$_ = id^ and P<2 t -p Xa (S) = P Xa (S)P2 
respectively P2-P Xa {T) = p Xa (T _1 )P 2 ,- and therefore P-2 t -JCt 
The permutation matrix Pi,- corresponds to the permutation A 2> _ ° o 
and hence is determined by the coset representatives Ji-Rjjo, 



C a. Xa p 2, 



Proof. For our choice of coset representatives Rj as given in (2.1.4) 
one finds for p Xa (S) 
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(4.0.31) 



(0 


1 











o\ 


1 





























g— 2nia 




















1 








g27ria 



















1 








and hence the matrix Q2- is given by 



(4.0.32) 



Q 2 



For p Xa (T) one finds 



(4.0.33) 



(0 










1 





0\ 



















1 
















g— 2ma 







1 

























g27rict 











\o 


1 













( V 




(e 


2iria 



























1 
























1 
























1 












1 






























e 


-I'KtOL 



A simple calculation then confirms that Pi~p Xa {S) = p Xa {S)P2- re- 
spectively P 2 -p Xa {T) 



(4.0.34) 



/o 






V 1 



with 


1 







1\ 









'V 



and hence defines a symmetry of the transfer operator L^^. The 
matrix P2- coincides with the permutation matrix P2 corresponding 
to the permutation <j 2 in (2.1.6). □ 

Remark 4.0.4. For the trivial character \o a ls° the map jo-Z = —z* 
defines an automorphism of the Maass forms for the group r (4), indeed 
this is an automorphism for all Hecke congruence subgroups T (N). In 
this case the permutation Ao,- is the trivial permutation and the matrix 
Qo,~ is determined by the permutation a oS. For To (4) this is given by 
(4.0.28). Using (4.0.31) with a = one obtains for P - = p X0 (S)Q - 
just the permutation <Ji as given in (2.1.5). The symmetry P\ for 
To (4) and trivial character xo hence corresponds to the automorphism 
z — > —z* of the Maass forms for this group. 
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We have seen that for every eigenfunction / = f(Q of the op- 
erator P 2 £~g Xa with eigenvalue A = ±1 the function \j£ = }&_(() = 
P2P Xa (T ~ 1 S)P 2 f(( — 1) fulfills the functional equation 

(4.0.35) 2(C) = AC 2/ V (S)P 2 3L(h = A J_2(C) 

and hence is an eigenfunction of the involution J_ corresponding to the 
automorphism j_ = j 2 _ of the Maass forms for T (4) and character 
Xa- Hence this shows 

Proposition 4.0.5. The eigenfunctions f = f(Q of the operator P 2 £p x 
with eigenvalue A = ±1 correspond to Maass forms which are even re- 
spectively odd under the involution J_ = J 2 ,— 

Phillips and Sarnak have shown in [18] for a conjugate character 
Xa that the Maass cusp forms odd under the corresponding conjugate 
involution J stay cusp forms under the deformation of this character. 
Hence we get corollary of their result 

Corollary 4.0.6. The zero's of the Selberg zeta function for the group 
To(4) and character Xa corresponding to eigenfunctions of the operator 
P 2 Ct with eigenvalue A = — 1 which for a = are on the critical 
line $1/3 = | stay for all a on this line. 

Remark 4.0.7. The operator P 2 £~p Xa can be used to calculate numer- 
ically the Selberg zeta function for small values of $s(3 and arbitrary 
< ol < |. These numerical calculations confirm the above Corollary 
and let us expect that all the zero's of the Selberg function correspond- 
ing to the eigenvalue A = 1 of the operator P 2 C~p for a = leave 
the critical line when a becomes positive. A detailed discussion of the 
numerical treatment of the behaviour of the zero's of Selberg's function 
under the character deformation will appear elsewhere [1]. 
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